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EXECUTIVE SUMMARY 

21 st Century Community Learning Center (21 st CCLC) grants are designed to support out- of- school time 
(OST) programs that provide academic support for youth attending high- poverty, underperforming 
schools. The programs also offer enrichment activities such as art and music, recreation, and career and 
technical education. Pennsylvania has funded 21 st CCLC programs since 1998, and the 20 11- 12 awards 
are Pennsylvania's sixth cohort of grantees. 

Of the Philadelphia organizations awarded 21 st CCLC grants, 11 OST providers selected Research for 
Action (RFA) as their local evaluator. 1 RFA conducted a mixed- methods evaluation that examined 
student demographics, elements of program guality, and important student outcomes. We report 
aggregated findings in this report, and provide a set of recommendations for program improvement. 


Student Participation 

• The 2 1 st CCLC programs enrolled students from underperforming schools. 

• The 2 1 st CCLC programs served a predominantly African-American and Latino student 
population. 

• Representation of students with limited English proficiency (LEP) and students with disabilities 
was lower in the OST programs than in the schools that they attended. 2 

• The maj ority of elementary school students participated in 2 1 st CCLC programs at a meaningful 
level (90 + days or half of the school year) . 

• The maj ority of middle and high school students did not attend at a meaningful level ( 90 + days 
or half of the school year) . 


1 Note: One OST provider served as an intermediary, subcontracting to three organizations who actually ran the OST programs. 

2 High schools were the exception. High school OST programs had a higher representation of LEP students than in the schools they served. 
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Elements of Program Quality 


A. Alignment of Program Content to Student Academic and Behavioral Goals and Intended 

Outcomes 

• Programs aimed to improve academic achievement through homework help and proj ect- based 
learning, which comprised a majority of program time. 3 

• Programs supplemented academic support with enrichment activities such as sports or arts, but 
these activities varied in the degree to which they were designed for intentional skill- building. 

• Programs were largely unable to provide a significant amount of one- on- one or small group 
academic support or align academic support to school day activities. 4 

B. Well-Prepared Staff 

• Providers typically employed staff with high levels of education, but staff were relatively new to 
working for their provider. 

• Staff reported competency in areas related to the delivery of high quality youth programs, such 
as developing relationships with youth and supporting homework help. 

• Staff reported behavior management as an area for growth and support through professional 
development. 

• Professional development was readily available to staff, but staff believed they could use 
additional support in student behavior management and parent and community 
communications. 

C. Robust School Partnerships 

• All 2 1 st CCLC providers had goals which aligned directly or indirectly with those of schools and 
principals. 

• The majority of OST providers reported that their relationships with principals were defined by 
mutual respect and consistent communication. 

• A maj ority of providers utilized overlapping or blended staffing models which helped to ensure 
continuity between the school day and OST program. 

• A few providers experienced challenges establishing trust and mutual respect with new 
principals. 

• There was significant variation among providers in the depth of the partnerships they developed 
with schools, particularly the degree to which they developed relationships with the classroom 
teachers of students in their programs. 


3 Learning activities in which students work together to investigate relevant guiding question or problem chosen by students. PBL culminates 
in projects that require students to use and develop new skills and content knowledge. 

4 Black, A.R, Somers, M. A., Doolittle, F., Unterman, R., and Grossman, J .B. (2009) . The Evaluation of Enhanced Academic Instruction in 
After- School Programs: Final Report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U. S. Department of Education; Patricia A Lauer; Motoko AMba; Stephanie B Wilkerson; Helen S 
Apthorp; etal. (2006). Out- of- School-Time Programs: A Meta- Analysis of Effects for At- Risk Students, Review of Educational Research , 76,-p. 
275.; Moss, M., Swartz, J., Obeidallah, D., Stewart, G., Greene, D. (2001). AmeriCorps Tutoring Outcomes Study. Abt Associates. Cambridge, 
MA. 
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Student Outcomes 

Research on OST programs has found that they can have an impact on school attendance; academic 
outcomes, including standardized test scores; and sodo- emotional outcomes. 5 21 st CCLC programs were 
designed to improve student outcomes in the areas of school attendance, academic performance, and 
student behavior. Our analyses examined the potential impact of OST programs on all of these outcome 
areas. After taking into account the preexisting differences in academic and behavioral performance 
and observed background characteristics, our analyses suggest: 

• Regular elementary school OST partidpants were less likely to have 10 or more unexcused 
absences than students who did not partidpate or partidpated less than 30 days of the 
programming. 

• Regular middle school OST partidpants were more likely to earn higher course grades in math 
or reading than students who did not partidpate or partidpated less than 30 days of the 
program. 

Furthermore, higher levels of participation were associated with better outcomes in many, 
but not all, behavioral and academic performance indicators. More spedfically: 

For elementary participants, high levels of partidpation are assodated with: 

• A smaller chance of having 10 or more unexcused absences or out- of- school suspensions 

• Fligher reading course grades 

• Fligher PSSA reading and math scores 

• A higher chance of reading at grade level for 1 st - 3 rd grade students 

For middle school participants, high levels of partidpation are assodated with: 

• A smaller chance of having 10 or more unexcused absences or out- of- school suspensions 

• Higher reading, math, and sdence course grades 

• Higher PSSA reading scores 

For high school participants, high levels of partidpation are assodated with: 

• A smaller chance of having 10 or more unexcused absences or out- of- school suspensions 

• A higher chance of earning all the credits attempted in ELA and math 

Recommendations 
A. For Program Providers 

• Focus on increasing partidpation, particularly for middle and high school students. 


5 Durlack, R & Weissberg R (20 12) . After- school programs that follow evidence- based practices to promote social and emotional 
development are effective. Expanded Learning and After-school: Opportunities for Student Success. 

http:/ / www.expandindeaminaorg/ docs/ Durlak&Weissbera FinaLpdf: Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School 
Programs in the 21 st Century; Their Potential and What it Takes to Achieve it. Issues and Opportunities in Out of School-Time Evaluation: 
Issue 10. Cambridge, MA: Harvard Family Research Project 


• Focus on increasing the percentage of program staff with bachelor's degrees and explore ways to 
increase the percentage of certified teachers for programs serving high school students. 

• Continue to develop relationships with school personnel, particularly teachers, to facilitate 
linking of program activities to school day activities. 

• Consider blended or overlapping staffing models and leverage their potential to provide 
additional opportunities for OST staff to build relationships with the school and ensure the 
alignment of the school day to the OST program. 

• Enhance STEM programming to ensure that students have the opportunity to participate in 
hands-on, inguiiy- based activities that are known to impact student interest, awareness, and 
achievement in STEM. 

• Consider ways to provide more one- on- one and small group tutoring efforts. 

• Consider engaging in reciprocal data sharing so OST providers can better target their individual 
or collective support for participants. 

• Focus on leveraging 2 1 st CCLC advisoiy councils to build staff competencies as staff reported 
that building relationships with parents and utilizing community resources were their most 
challenging tasks. 

B. For Philadelphia’s OST System 

Continue to support citywide professional development of OST staff in the following 
areas: 


• Build the capacity of providers to support their front-line staff with behavior management 
through train the trainer professional development or other resources. 

• Support providers in identifying ways to build skills and knowledge related to their 
neighborhoods, community resources and parents. 

• Provide professional development and coaching on best practices to implement hands- on and 
inguiry based activities for providers that have STEM programming. 

• Support providers in thinking about how to better engage middle and high school students. 

Work with the School District of Philadelphia (SDP) to develop school-level support for 
OST-school partnerships. The SDP could strengthen OST programs in the following ways: 

• Encourage principals to deepen the partnerships between OST and school staff and create 
opportunities for further communication between OST staff and teachers. 

• Encourage schools to have common professional development or common planning time with 
their OST providers about the curriculum and the academic improvement needs of their students. 

• Make student data accessible to program providers in real time so that they can target academic 
interventions for program participants. 
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Introduction 

21 st Century Community Learning Center (21 st CCLC) grants are designed to support out- of- school time 
(OST) programs that provide academic support for youth attending high- poverty, underperforming 
schools. The programs also offer enrichment activities such as art and music, recreation, and career and 
technical education. Pennsylvania has funded 21 st CCLC programs since 1998, and the 20 11- 12 awards 
are Pennsylvania's sixth cohort of grantees. 

Pennsylvania has funded 21 st CCLC programs since 1998, and awarded two rounds of grants in 2011- 12 
to its sixth cohort of grantees. The first round of Cohort 6 grants focused primarily on providing 
academic and enrichment programming for elementary school students while the second round (Cohort 
6A grants) focused primarily on providing middle school STEM and high school credit recovery 
programming. 

Of the Philadelphia organizations awarded 21 st CCLC grants, 11 organizations representing 12 OST 
providers selected Research for Action (RFA) as their local evaluator. 6 As part of a citywide effort to 
create a cohesive system for OST programs, RFA agreed to aggregate the data from these local 
evaluations to provide an overview of OST programming and student outcomes in Philadelphia. While 
these providers represent a third of all Cohort 6 and 6A 21 st CCLC grantees in Philadelphia, the analysis 
and results presented in this report provide useful insights into the potential impact and needs of OST 
programs in Philadelphia and help inform the development of Philadelphia's OST system- building 
efforts. 

About this Report 

This report examines the 2 1 st CCLC programs evaluated by RFA and presents an aggregate analysis of 
key findings to inform broader discussions about OST programs. The report addresses the following 
guestions: 

1. Who participated in the 2 1 st CCLC programs and how freguently did they participate? 


6 One grantee was an intermediary organization representing two provider organizations. 
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2 . What organization- level elements of quality programming were in place to achieve the intended 
outcomes? Specifically: 

• What was the content of the 2 1 st CCLC programs and how did the content support 
program goals for academic and behavioral improvement as well as STEM interest and 
engagement? 

• What were the characteristics of staff who implemented the 2 1 st CCLC programs and 
what types of organizational supports were available to support them, particularly 
for those implementing STEM programming? 

• What was the reported level of partnership between 2 1 st CCLC programs and 
the schools their students attended? 

3. What was the relationship between program participation, organization- level elements of 
quality programming and academic, attendance and behavioral outcomes? 

RFA addressed these research questions through a mixed- methods study that analyzed the following: 

• Student outcomes data 

• Program participation data 

• OST staff survey responses 

• Qualitative data from local evaluations, including interviews with program staff, site visits, and a 
review of program documents 

The study includes data from each of the 50 sites operated by the 11 grantee organizations. However, 
the analysis of student outcomes focuses on a smaller subset of 40 sites from 11 providers which 
operated programs in public or charter schools. Data from Catholic schools could not be integrated into 
the student outcome analyses because these schools utilize different grade structures and assessments 
(TerraNova rather than PSSA scores). 7 In addition, the analysis of the organization- level elements of 
quality programming focuses only on 10 providers with staff who completed the staff survey. 

Student outcomes analyses considered the following: 

• Math, reading, and science course grades 

• Math and Reading Pennsylvania System of School Assessment (PSSA) scores 

• Developmental Reading Assessment ( DRA) for 1 st - 3 rd grade students 

• HS credits earned for 9 th - 12 th grade students 

• School attendance and suspensions 

Student outcomes analyses compared OST regular participants to students who did not regularly 
participate in the OST programs and non- participating students within their schools. The analysis also 
considered the level of program participation and the presence of key organization- level elements of 
quality programming. A more detailed description of the data collected and methodology can be found 
in Appendix B and C. 


7 A separate analysis for these Catholic schools is available in Appendix D. Catholic Social Services also operated two community centers which 
drew from public schools. CSS programs serving public school students are included in the aggregate analysis. 



Overall, this study found promising evidence of the benefits of the 2 1 st CCLC programs in two outcome 
areas— school attendance and course grades— for regularly participating elementary and middle school 
students. In addition, for students who participated in the program, the analyses found that outcomes 
in most areas improved as program participation increased. In addition, staff survey data revealed key 
organization- level elements of guality programming that may be related to more positive student 
outcomes. Our analysis of 21 st CCLC program content, staffing, and school partnerships found variation 
in the implementation of these organizational- level elements of guality programming and suggests ways 
in which programs could increase their impact on students' outcomes. 

The report is organized into the following main sections: 

I. Context: 21 st CCLC Providers, Schools, and Programs 

II . Participant Characteristics and Level of Participation 

III. Organization- Level Elements of Quality Programming: Promising Practices and Areas for 
Growth in Program Content, Staff, and School Partnerships 

IV. Student Outcomes and the Relationship between Student Outcomes with Level of 
Participation and Organization- Level Elements of Quality Programming 



I. Context: 21 st CCLC Providers, Schools, and Programs 

21 st CCLC programs operate in a variety of settings including non- profits, university based centers, and 

charter schools. See Table El in Appendix E for a full list and profiles of provider organizations. 

Overall, providers served a diverse array of schools. 

• The 50 sites in which 2 1 st CCLC programs operated included 46 schools and four community 
centers that served multiple schools. 

• Programs typically drew participants from the schools in which the programs were located. 
However, the four community centers drew from several schools in their geographical areas. For 
the purposes of this study, we asked these programs to identify one primary feeder school. 

• The schools included in this study were traditional public schools, charter schools, and Catholic 
schools. 

• Schools included elementary (K-5 and K-8), middle, and high school grades. However, providers 
did not always serve all grade levels in the schools. 

• Overall, the 2 1 st CCLC programs served students in schools that were underperforming and had 
student populations that were predominantly low- income students of color. 8 

Table 1 displays the number and type of schools in the study. 


Table I . Profile of Schools Served by 2 1 st CCLC Cohort 6 Grants 


ELEMENTARY (K-5) 

444444444410 

ELEMENTARY (K-8) 

4444444444444444444 1 9 

MIDDLE (5-8) 

4444444444444444 1 6 

HIGH SCHOOL (9- 12) 

444445 

TOTAL 

mtttmmtttmt 

444444444450 



Traditional Public School Charter School / Catholic School 


Data was not available to determine the performance of the Catholic schools in the study prior to 21 st CCLC implementation. 


21 st CCLC programs were located throughout the city; the highest concentration was in 
North Philadelphia. Figure 1 displays the locations of 21 st CCLC provider headquarters and program 
operation sites. 

Figure I. Neighborhoods of 2 I st CCLC Programs 

Provider 

• Catholic Social Services 

• CityYear 

• Communities in Schools 
« Congreso 

Education Works 

• Netter Center 

• PAE P 

• PH MC 

• Pan American Charter School 

• Sunrise 

• TOC FWH 



• 21 st Century programs evaluated by RFA operated in 38 of the 158 Philadelphia neighborhoods. 

• Programs were concentrated in North Philadelphia. 

Figure 2 identifies the neighborhoods that had more than one RFA evaluated 21 st CCLC program. 


Figure 2. Neighborhoods with More Than One RFA-Evaluated 21 st CCLC Program 


Number of Sites by Neighborhood 


lit 


1 1 1 1 1 
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<Nr 


<c^ x </° .J?' 
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^ ^ ^ fj? 


<c v 


^ </' 


• 30 neighborhoods had one 2 1 st CCLC program. 

• Eight neighborhoods had two or more 2 1 st CCLC programs. Fairhill and Elm wood hosted four 
programs. 

Program content varied by provider. Providers offered a range of program content including arts 
programs, STEM programs and sports activities. Programs also varied by grade level. However, there 
were also many similarities in program content, particularly regarding academic supports. 

Table 2 illustrates key program similarities and differences in program content among providers. 

Table 2. 21 st CCLC Program Content: Key Similarities and Differences 


000 


ooo 

SIMILARITIES 


DIFFERENCES 

• All providers offered homework help. 

• More than three quarters of programs offered 
project-based learning. 

• Seven of eleven providers offered additional 
academic enrichment programming including math 
and literacy. 

• All offered other enrichment activities in the areas 
of arts, service learning, culinary, and other areas. 

• All but two offered physical activity as part of 
programming. Examples include organized sports, 
dance, cheerleading, and gym time. 


• Six of the eleven providers included STEM activities 
in their programs such as robotics, nutrition or 
recycling projects or Fantasy Baseball competitions. 

• High school programs offered homework help on an 
“as-needed basis” while it was a required 
component of elementary school and middle school 
programs. 

• Few programs (seven out of 50) offered tutoring on 
a regular basis. 

• Few programs (five out of 50) offered college 
preparation or college-themed activities. 

• Only three of 50 programs had character 
development or mentoring activities. 

• High school programs were less likely to offer 
physical fitness activities. 
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II. Participant Characteristics and Level of Participation 

Level of participation in OST programs is a key determinant of the benefits students receive from the 
programs. 9 In this section, we examine which students were participating in the 2 1 st CCLC programs 
and their levels of participation. 


A. Participant Characteristics 

21 st CCLC programs are designed to serve youth in high- poverty, underperforming schools. Research on 
OST program participation across the country has found that this population of students has less access 
to and participates less freguentiy in OST programs. 10 

21 st CCLC participants came from traditional public, charter, and Catholic schools. Figure 
3 displays the percentage of students enrolled in the 2 1 st CCLC programs by the type of school they 
attended. 

Figure 3. Percentage of Students Enrolled in 2 I st CCLC Programs by School Type 



n=50 sites 


• A total of 4, 594 students were enrolled in the programs for which we had participation data. 

• More than 80 % of participating students were enrolled in traditional Philadelphia public 
schools and approximately 4% were in charter schools. 

• About 14% of participating students were enrolled in Catholic schools. 

21 st CCLC participants were predominantly African-American and Latino. Limited English 
speakers and students with disabilities were under-represented in comparison to the populations in 
participating schools. Figure 4 displays the demographic characteristics of OST participants in 
comparison to the demographics of all other students in the same schools. 


9 Little, P., Wimer, Christopher, Weiss, H. B. (2008) . After School Programs in the 21 st Century: Their Potential and What It Takes to Achieve 
It. Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project 

10 Ibid. 
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Figure 4. Demographic Characteristics of 2l st CCLC Participants and Non-OST Students 


OST PARTICIPANTS 


NON-OST PARTICIPANTS 


in the primary feeder schools 






African 

American 



Latino 



White 



Asian 



Other 



Male 



Female 


FRL: 

Free and 
Reduced 
Lunch 

LEP: 

Limited 

English 

Proficiency 


SWD: 

Student 

With 

Disability 


• Race/ Ethnicity: Overall, the programs in our study served a predominantly African-American 
and Latino student population. African-American students represented 52% of students at the 
elementary school level, 72% of students at the middle school level, and 66% of students at the 
high school level, while Latino students represented 33%, 22%, and 28% of elementary, middle 
and high school participants, respectively. Compared to the general student population at their 
schools, African-American students were over-represented in OST programs. 

• Free and Reduced- Price Lunch: Roughly 83% of OST elementary and middle school 
participants qualified for FELL, and 73% of high school participants qualified for FRL. Overall, 
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the percentage of OST participants qualified for FRL was similar to the general population at 
their schools. 11 

• Limited English Proficiency: Thirteen percent, 9%, and 12% of the elementary, middle, and 
high school OST participants respectively were identified as LEP, which is smaller than the 
general student population at the schools they attended in 20 12- 13. 

• Students with Disabilities: Nine percent, 18%, and 15% of the elementary, middle, and high 
school OST participants respectively were SWD. The percentages are smaller than the general 
student population at the schools they attended in 20 12- 13. Fifty- six percent of high school OST 
participants were SWD, while 63% of non- OST participants were SWD. 

• Male/ Female: At high school level, male students were slightly underrepresented in OST 
programs as compared to the non- OST population in the feeder schools. 


B. Participation Levels 

Research has consistently identified a relationship between a student's level of participation in an OST 
program and the benefits received. No consistent level of overall program "dosage" has been identified 
for achieving the benefits of OST programs. However, at least two studies found academic benefits for 
students who participated in programming between 60-90 days. 12 In this report, we use 30 days as the 
minimum level of participation because 30 days is the cut-off used to define a program participant for 
federal 21 st CCLC reporting requirements. We use 90 days as the threshold for the upper range of 
participation because it matches the dosage level reported in a 2004 evaluation of 21 st CCLC programs 
that resulted in positive academic outcomes for students. In this 2004 evaluation, participants showed 
slightly increased scores on the Iowa Test of Basic Skills (ITBS) after 60 days but a "stronger impact" on 
the ITBS score after 90 days. 13 

A majority of K-5 students participated 90 or more days, while more than half of middle 
and high school students participated less than 30 days. Figure 5 illustrates these participation 
levels by grade levels in 20 12- 13. 


11 Free and reduced priced lunch (FRL) is a problematic indicator of income status because families are required to submit paperwork to 
qualify for this status and can cause FRL to be under-reported. However, FRL is the only indicator of socio-economic status available througb 
the School District of Philadelphia 

12 Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School Programs in the 21 st Century: Their Potential and What it Takes to Achieve 
it. Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project; Black, A. R, 
Somers, M.-A., Doolittle, F., Unterman, R, and Grossman, J. B. (2009). The Evaluation of Enhanced Academic Instruction in After- School 
Programs: Final Report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, United States Department of Education. 

13 Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School Programs in the 21 st Century: Their Potential and What it Takes to Achieve 
it. Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project, page 6. 
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Figure 5. Percentage Enrollment by OST Participation and Grade Level 



Note: n=4, 337; 257 participants are not reported in this figure because grade levels were not available in the data. 


• More than 80% of elementary school students, including K- 2 and 3 rd - 5 th grade students, 
attended more than 30 days, and more than half of them attended 90 days of the 21 st CCLC 
programs. 

• J ust over half of middle school students attended fewer than 30 days of the OST programming. 

• More than two- thirds (68%) of high school participants attended fewer than 30 days. 


C. Section Summary 

The 21 st CCLC programs enrolled students from underperforming schools, and more than two- thirds 
gualified for ERL (an indicator of poverty) . However, representation of students with LEP and SWD 
was somewhat lower in the OST programs than in the schools that they served. The majority of 
elementary school students participated in 21 st CCLC programs at a meaningful level (90+ days), yet the 
majority of middle and high school students attended less than 30 days of the programming, possibly 
weakening the effect of the supports received through the OST program. 
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III. Organization-Level Elements of Quality Programming: Promising 

Practices and Areas for Growth in Program Content, Staff, and School 
Partnerships 

21 st CCLC funding is awarded to programs with the goal of improving students' academic outcomes 
(e.g., standardized test scores and grades), as well as students' school attendance and behavior. 
Research identifies several organization- level elements of guality programming that contribute to 
implementation of high- guality, academically- focused OST programming. These are: 

1. The alignment of program content with student academic and behavioral goals; 14 

2 . The training and expertise of the staff who deliver program supports; 15 and 

3. Strong partnerships with schools to inform the design of program content and other 
supports provided by staff. 16 

In this section, 17 we examine the extent to which these organization- level elements of guality 
programming were in place for the 2 1 st CCLC Cohort 6 and 6A programs. 18 


A. Organization-Level Element of Quality Programmin85/9g: Alignment of Program Content to 
Student Academic and Behavioral Goals and Intended Outcomes 
Research suggests that OST programs can impact academic achievement, school attendance, and 
behavior when the programs: 

• Set clear and consistent goals for academic improvement 19 

• Provide academic support that includes: 

o A meaningful amount of support; 20 
o Small- group or one- on- one settings; 21 

o Alignment with the school curriculum or grade- specific academic standards, 22 and 
o A meaningful amount of support for program design and planning. 


14 Iittle, P., Wimer, Christopher, Weiss, H. B. (2008) . After School Programs in the 21 st Century; Their Potential and What it takes to Achieve it. 
Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project 

15 Huang, D., Dietel, R. (2011). Making after- school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California.; Nee,J. 
(20 11). Core knowledge and competencies for Afterschool and Youth development professionals. National Institute for Out-of- School Time: 
Wellesley Center for Women. ; Bodilly, SusanJ. and Megan K. Beckett. Making Out-of- School- Time Matter: Evidence for an Action Agenda. 
Santa Monica, CA: RAND Corporation, 2005. http://www.rand.org/pubs/monographs/MG242. Also available in print form.; Starr, E., 
Gannett, E.„- Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School Programs in the 21 st Century; Their Potential and What it takes 
to Achieve it. Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project 

16 Huang, D., Dietel, R (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California 

17 Please see the Appendix for more details on the methodology that guided the development of this section of the report. 

18 Note: We were not able to conduct an assessment of program implementation. 

18 Huang, D., Dietel, R (2011). Making after- school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California 

20 Black, A. R., Somers, M.-A., Doolittle, F., Unterman, R, and Grossman, J . B. (2009) . The Evaluation of Enhanced Academic Instruction in 
After- School Programs: Final Report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education; Patricia A Lauer; Motoko AMba; Stephanie B Wilkerson; Helen S 
Apthorp; etal. (2006). Out- of- School-Time Programs: A Meta- Analysis of Effects for At-Risk Students. Review of Educational Research , 76,-p. 
275.; Moss, M., Swartz, J., Obeidallah, D., Stewart, G., Greene, D. (2001). Americorps Tutoring Outcomes Study. Abt Associates. Cambridge, 
MA. 

21 Ibid. 

22 Huang, D., Dietel, R (2011). Making after- school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California. 
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In addition, research suggests that STEM programs can influence student interest, engagement, and 
even academic achievement in STEM when they include hands on learning environments. 23 

Promising Practices: Alignment of Program Content to Student Academic Goals and Intended 
Outcomes 

Across 21 st CCLC providers, programs supported academic improvement in one or more of the 
following ways: 

Academic goals were consistent among providers. Research indicates that setting clear and 
consistent goals for OST programs that are related to academic improvement can positively impact 
student academic achievement as well as school attendance and behavior. Providers most often 
reported that their primary goal was to improve academic achievement, social skills, and behavior. 
Improving school attendance was seldom mentioned as a primary goal. Additional goals identified by 
most providers included increasing student motivation, self-confidence and self-efficacy. 

STEM goals were well-aligned to indicators of high quality programming. Six of the 

providers included STEM activities as a core or supplemental component of their programming. 
Research indicates that high quality STEM OST programs typically help students develop a deeper 
interest in STEM, develop an awareness of STEM career opportunities, and increase their STEM 
content knowledge 24 . Aligned with the research, the top STEM goals most consistently dted across 
providers included increasing student interest in STEM and their awareness of STEM careers and 
improving STEM student achievement. 

Programs provided a meaningful amount of academic support through homework help 
and/or project-based learning (PBL). According to research, programs demonstrating 
achievement level gains in math and reading provide between 1.5 to 3.5 hours of academic support each 
week, depending on the structure of the support (e.g., one study of one-on-one support demonstrated 
impact on literacy outcomes with 1.5 hours of tutoring per week) . 25 Most providers devoted between 30- 
60 minutes per day, or 2- 5 hours per week, to homework help— a meaningful amount of support as 
suggested by the research. In addition to homework help, providers offered other types of academic 
enrichment including academic games, such as Scrabble, spelling bees, fantasy baseball or STEM 
activities. Finally, all elementary school providers offered 30-90 minutes of PBL three- to- five days per 
week. PBL is intended to incorporate academic skills but providers' PBL activities varied in the extent to 
which they incorporated these skills. 

Programs offered enrichment activities, but these varied in the amount of intentional 

skill-building. Effective OST programs augment academic support with other types of enrichment 
activities focused on skill- building. 26 All 21 st CCLC programs offered enrichment activities each week 
and all but two programs offered various types of physical activity as well. However, enrichment 
activities varied in the degree to which they incorporated intentional skill- building. 


23 Campbell et al., 2002; Davis and Rosser, 1996; Froschi et al., 2003; Hansen, et al.,1995; Koch, 2002; Lee, 1997; Wenglinsky, 2000 

24 Afterschool Alliance (2013). Definingyouth outcomes for STEM learning in afterschool. Retrieved from: 
http://www.afterschoolalliance.org/ S1EM Outcomes 20 13.pdf 

25 Moss, M., Swartz, J., Obeidallah, D., Stewart, G., Greene, D. (2001). Americorps Tutoring Outcomes Study. Abt Associates. Cambridge, MA. 
26 Durlack, R &Weissberg, R (2012). After-school programs that follow evidence-based practices to promote social and emotional 
development are effective. Expanded Learning and After-school: Opportunities for Student Success. 

http:// www.expandincdeamina.org/ docs/ Durlak&Weissbera Final.pdf 
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Areas for Growth: Alignment of Program Content to Student Academic Goals and Intended Outcomes 
Across providers, the alignment of program content with desired student academic outcomes was 
limited in a few ways. 

Fewer than half of the programs offered subject-specific academic support in a one-on- 
one or a small group setting . 27 OST programs that have demonstrated impact on academic 
outcomes provide one-on-one or small group tutoring focused on particular subjects like math or 
English or provide explicit instruction in those subject areas to fewer than nine students at a time. 
Across the 50 programs, only three elementary, three middle, and one high school program offered 
explicit tutoring or direct, subject- specific instruction in this format. 

The observed STEM program implementation did not align with known best practices. 

Research indicates that high quality STEM programs include opportunities for students to engage in 
hands-on, inquiry- based learning opportunities; encourage a high level of participant engagement 
through purposeful activities; and are driven by STEM content knowledge and practices. Program 
observations revealed that, although staff believed they saw student engagement in the program, the 
activities did not include many of these known indicators of high quality STEM practices. Inquiry and 
reflection come from activities that are student directed and have significant opportunities for students 
to explore, question, and discover answers primarily on their own. Our observations suggest that these 
types of activities were not generally integrated into STEM programming at provider sites. 

Support for program design and planning varied. Support for program design and planning was 
similar across providers in that planning for daily program activities typically happened at the site 
level— the majority of provider staff survey respondents reported that either the group leaders or the 
“staff team" planned daily activities. However, organization- level support for planning varied. A 
majority of staff across all providers reported receiving feedback and guidance on daily lesson plans and 
a majority of staff at eight of ten providers reported receiving templates for lesson planning. Fewer 
providers (5/10) made curriculum available. However, STEM programs often drew upon curriculum. 
Even fewer providers (2/10) had a majority of staff reporting receiving paid planning time which may 
be needed, particularly if curricular materials are not available. Across providers, group leaders were 
more likely than site coordinators to plan daily activities but they were consistently less likely to have 
paid planning time. Group leaders averaged four hours/ week of paid planning time while site 
coordinators averaged 11 hours/ week of paid planning time. Thus, there seemed to be some 
discontinuity between who had dedicated/ paid time for planning (site coordinators) and who actually 
planned the activities (group leaders) . 

Alignment to the school-day curriculum was not consistent across providers. Only four 
providers reported some alignment between the school-day curriculum and after- school activities. The 
programmatic model for two providers called for using the schools' curricular topics to develop aligned 
enrichment activities. In the other two instances, having a blended staffing model— teachers or 


27 Black, A. R, Somers, M.-A., Doolittle, F., Unterman, R, and Grossman, J . B. (2009). The Evaluation of Enhanced Academic Instruction in 
After- School Programs: Final Report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U. S. Department of Education; Patricia A Lauer; Motoko AMba; Stephanie B Wilkerson; Helen S 
Apthorp; etal. (2006). Out- of- School-Time Programs: A Meta- Analysis of Effects for At- Risk Students. Review of Educational Research . 76,-p. 
275.; Moss, M., Swartz, J., ObeidaUah, D., Stewart, G., Greene, D. (2001). Americorps Tutoring Outcomes Study. Abt Associates. Cambridge, 
MA. 
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paraprofessionals from the school day work in the OST program along with other OST staff— or a strong 
school day presence may have helped the provider align programming to the school day. Even when 
incorporating significant academic content to their programs, most providers did not reference the 
school curricula in planning those activities. 

B. Organization-Level Elements of Quality Programming: Well-Prepared Staff 

Skills and expertise of OST staff play a major role in whether youth become engaged in a program and 

whether the program is beneficial to them. Research has found that: 

• Highly gualified OST staff typically have these characteristics: 

o experience in the OST field 
o experience working with the provider 
o high levels of education 28 29 

o demonstrable competencies in a number of areas deemed important for youth 
development professionals. 30 

• STEM providers should have content knowledge in a STEM field 31 . 

• Well- designed professional development can support the development of highly- effective OST 
staff. 32 

Promising Practices: Well-Prepared Staff 

As noted, research indicates that highly qualified staff members typically have experience in the OST 
field, experience working with the OST provider, and high levels of education. RFA's staff survey 
analysis, as well as qualitative data and analyses, found that OST providers are increasingly hiring staff 
with these research- based qualifications. 

At least 50% of staff at more than half of the providers have a college degree and 5+ years 
of OST experience. Table 3 illustrates the number of providers with 50% or more of their OST staff 
having the research based qualifications of experience in the OST field, experience working with the 
OST provider, and high levels of education. 


28 Huang, D. & Dietel, R (20 11). Making Afterschool Programs Better. National Center for Research on Evaluation, Standards, & Student 
Testing 

29 Bodilly, SusanJ. and Megan K. Beckett. Making Out- of- School-Time Matter: Evidence for an Action Agenda. Santa Monica, CA: RAND 
Corporation, 2005. http://www.rand.org/pubs/monographs/MG242. Also available in print form. 

30 N10ST (2012) Retrieved from: http://www.wallacefoundation.org/knowledge-center/after-school/quality-and-cost/Documents/Strong- 
Directors-SMlled-Staff-Guide-to-Using-the-Core-Competencies.pdf; OSIRC (2010). OST Staff Competencies. 

31 A Review of the Literature and the INSPIRE Model STEM in Out- of- School Time (J une 2007) 

32 While research on the effectiveness of professional development on improving program quality is inconsistent. Several studies have found 
benefits for program staff and programs. (New York State After- School Network (2011). After-school professional development; Resources, 
Outcomes and Considerations; Out-of- School Time Resource Center Summaiy of literature: Evaluating the Impact of Teacher Trainings on 
OST Program Quality (2003). 
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Table 3. Number of Providers with 50% or More of Staff with College Degrees, 5+ Years with Provider and In OST Field 

Number of Providers 


50% or more staff with 
5+ years in OST 

50% or more staff with 
5+ years with provider 

50% or more staff have 
college degrees 





2 


2 


3 


3 




5 


5 


6 


6 



8 


Providers hired staff with college degrees. Research indicates that there is a relationship 
between the level of staff education and quality programming. 33 Staff of ten of the eleven providers 
completed the staff survey. 

• Eight of ten providers had 50% or more staff with college degrees. 

• Seven of ten providers had 75% or more site coordinators with college degrees. 

• Five of ten providers had 75% or more group leaders with college degrees. 

• Two of ten providers also employed classroom teachers to staff their programs. 

Staff report competency in areas related to the delivery of high quality youth programs. 

The staff survey asked respondents to identify their top strengths from a list developed by the National 
Institute on Out of School Time (MOST) as well as the Out of School Time Resource Center (OSTRC) at 
the University of Pennsylvania. These competencies are known to be related to positive OST youth 
outcomes. 

The following are the most commonly reported strengths, across providers: 

• Developing positive relationships with program participants ( 8 of 10 providers) 

• Supporting homework help ( 7 of 10 providers) 

• Presenting program activities in an engaging way ( 5 of 10 providers) 

• Creating a safe environment ( 5 of 10 providers) 

These survey findings are encouraging because research shows that "youth engagement is key to 
positive outcomes for youth in afterschool programs. " 34 In addition, the ability to develop positive 
relationships with youth and the ability to create safe spaces for youth are important OST staff 
competencies. 35 

Staff participated in significant professional development opportunities. Although there is 
no conclusive evidence on the impact of OST staff professional development for youth outcomes, 
professional development does matter for classroom teachers. 36 A review of over 1,000 studies on the 


33 Massachusetts After- School Research Study (2005). Pathways to Success forYouth: What Counts in After- School. 2005 United Way of 
Massachusetts Bay. 

34 Ibid. 

35 National Institute on Out of School Time (20 12) . Strong Directors and Skilled Staff: Guide to Using the Core Competencies. Out of School 
Time Resource Center (2010). Philadelphia OST Staff Competencies and Content Areas. 

36 Hill, S. (20 12) . Leap of Faith: A literature review on the effects of professional development on program quality and youth outcomes. 
National Institute on Out- of- School Time. Commissioned by the Robert Bowne Foundation. 
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impact of professional development for classroom teachers revealed that providing teachers with more 
than 14 hours of professional development had a positive and significant impact on student 
achievement. 37 Research also indicates that well- designed professional development may support the 
growth of a highly- effective OST staff. 38 We did not collect information on the types of professional 
development opportunities in which staff participated. However, we did ask staff how much time they 
spent in professional development. On average, staff received about 33 hours of professional 
development through the provider and external sources each year— well above the 14 hours designated 
as meaningful in the literature. However, it is unclear whether all of these hours were spent on OST- 
spedfic professional development. 

Areas for Growth: Well-Prepared Staff 

The following section provides insight into the areas for staff growth across the eleven providers. These 
findings draw primarily from the staff survey but are also supplemented by gualitative data and 
analyses. 

Overall, 21 st CCLC program staff members were relatively new to working with their 
providers and in the OST field. 

• Only two providers had 75% or more staff with 5+years of OST experience. 

• The maj ority of providers had 2 5% or fewer staff that had been working with the provider for 5 
years or more. 

Overall, the majority of staff delivering STEM programming did not have STEM 
backgrounds. Even though the research is mixed on whether staff that have strong STEM content 
background actually make a difference in STEM OST programs, staff with STEM backgrounds may have 
more capacity to align program content and design with school curricula and STEM OST program goals. 

• On average, providers reported that 32% of staff that led math activities had a STEM 
background. 

• On average, providers reported that 28% of staff that led science activities had a STEM 
background. 

• None of the six OST providers that included STEM activities in their programming had more 
than 47% of staff with STEM majors and the average across providers was 35%. 

• Only two of the six providers intentionally hired staff because of their STEM backgrounds. 

Staff reported challenges in several youth development competency areas. In the staff 
survey, across ten of the eleven providers, staff reported the following as top challenges or professional 
development needs, as reported in aggregate at the provider level: 

• Effectively guiding and managing student behavior ( 10 of 10 providers) 

• Identifying community resources and building community connections ( 7 of 10 providers) 


37 Yoon, Duncan, Lee, Scarloss, & Shapley (2007) .Reviewing the evidence on how teacher professional development affects student 
achievement. Institute of Education Sciences. 

38 While research on the effectiveness of professional development on improving program quality is inconsistent. Several studies have found 
benefits for program staff and programs. (New York State After- School Network (2011). After- School Professional Development: Resources, 
Outcomes and Considerations; Out-of- School Time Resource Center Summary of Literature: Evaluating the Impact of Teacher Trainings on 
OST Program Quality (2003). 
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• Building relationships with parents (6 of 10 providers) 

• Encouraging youth voice or leadership in the program (4 of 10 providers) 

• Creating and sticking to a structure in the program (4 of 10 providers) 

Identifying community resources, building community connections, and building relationships with 
parents are known OST staff competencies for high guality programming. Our analyses identified them 
as persistently challenging for staff and, therefore, in need of further attention by providers. 

C. Organization-Level Elements of Program Quality: Robust School Partnerships 
The strength of an OST program's partnership with its host school may contribute to the program's 
impact on academic outcomes, including participants' level of "homework effort" and homework 
completion 39 and even grades and test scores 40 . Strong school partnerships have also been related to 
higher levels of school attendance among program participants 41 . 

Research has identified six characteristics of strong school partnerships: 

1. Shared vision and goals: OST program staff and school staff share similar goals for 
program participants. 

2. School principal support: The program's relationship with the principal is 
characterized by mutual respect and regular, scheduled communication. 

3. OST staff are present in the school during the school day: OST programs may use 
a "blended staffing" model in which the schools' teachers or paraprofessionals are hired to 
work in the OST program. Alternately, OST programs could structure staff positions so 
that half or more of their work hours overlap with the school day and they are able to be 
present in the school building during the day. 

4. Multi-level partnerships: OST staff have relationships in the school with staff other 
than the principal. It is particularly beneficial for OST staff to have regular communication 
with classroom teachers. OST staff may also participate in school committees and/ or 
school staff meetings. Cooperation with the school's custodial, cafeteria and security staff 
is also important. 

5. Data access and use: OST program staff have regular and reciprocal sharing of 
information and data about student progress with school staff. 

6. Alignment to school behavioral norms: OST program staff negotiate behavioral 
expectations for program participants with the school to align with school norms while 
remaining appropriate for an OST program. 

Our analyses found that the 2 1 st CCLC programs in our study tended to be stronger in some school 
partnership areas than in others. Figure 6 displays the variation in these characteristics, as suggested by 
our data. 


39 Massachusetts After- School Research Study (2005). Pathways to Success for Youth: What Counts in After- School. 2005 United Way of 
Massachusetts Bay. 

40 Huang, D. &Dietel, R (2011). Making Afterschool Programs Better. National Center for Research on Evaluation, Standards, and Student 
Testing. 

41 Russell, C. A., &Reisner, E. R. (withj ohnson, J . C., Rouk, U., &White, R N.). (2005). Supporting social and cognitive growth among 
disadvantaged middle-grades students in TASC after-school projects. Washington, DC: Policy Studies Associates. 
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Figure 6. Evidence of Partnership Characteristics 42 
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As can be seen in Figure 6, most of the characteristics of strong school partnerships were observed for 
at least half of the providers or school sites in our sample. The following section describes providers' 
efforts to build strong school partnerships, including promising practices and areas for growth. 

Promising Practices: Robust School Partnerships 

Several characteristics of strong school partnerships were prevalent among providers and sites in our 
study. 

All 21 st CCLC providers had goals aligned with those of schools and principals. Research 
suggests that in order for providers to partner effectively with schools, they must address academic 
improvement. 43 As mentioned in the previous section, 21 st CCLC providers had clear and consistent 
goals for improving academic achievement, motivation to learn, and behavior. Our data confirmed that 
these goals are aligned to those of principals at OST school sites and that principals valued the OST 
programs. All but one principal expressed appreciation for the robust enrichment activities offered by 
the OST programs, including arts and music activities, which were not typically offered during the 
school day. Forty-one percent (41%) of interviewed principals (N= 17) also reported valuing the OST 
program because it provided a free, safe and productive environment after school, which helped the 
working parents of their school community. These principals appreciated that the program was not 
simply "babysitting" but involved skill- building and exposure to new activities. 

The majority of OST providers reported that their relationships with principals were 
defined by mutual respect and regular communication. The conditions of a supportive 
principal relationship - mutual respect and regular communication— appeared to be in place for the 
majority of OST- school partnerships in our study. Providers reported that regular communication with 
principals was occurring, more likely informally than formally. A majority of site coordinators reported 

42 Note: Providers have multiple school sites and the analysis rated these characteristics present if it appeared to be present for a majority of 
sites, based on staff and principal interviews. Or, if a maj ority of staff responded positively to relevant questions on the staff survey. 
43 Deschenes, S. N., Arbreton, A., Little, P.M., Herrera, C., Grossman, J.B., Weiss, H.B., Lee, D. (2010). Engaging older youth: Program and 
city-level strategies to support sustained participation in out- of- school time. Harvard Family Research Project. 
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that they communicated informally with the principal at least once per week via email, text, phone calls, 
or impromptu check-ins. Formal meetings with principals occurred less frequently, but seemed to occur 
at least monthly for eight of 11 providers. In some cases, principals were described as particularly 
invested. Activities of invested principals included the following (see Figure 7) : 


Figure 7. Principal Support Activities 


Contributed ideas for program planning 


! principals 


Reported to “trust” OST staff to do their job and did not micro-manage 


] principals 


Visited the program 


principals 


Promoted the program among teachers and parents 


principals 


Invited OST staff to the school meetings or events 


principals 


Advocated for the program internally and externally 

principals 


Provided input on hiring OST staff 

principals 


A particularly promising sign is that about 17% of principals were described as being involved in three 
or more of the above tasks and communicating daily or weekly with OST program staff. These positive 
and supportive relationships may have resulted, in part, from providers' long-term relationships with 
many of the schools receiving 21 st CCLC support. Thirty- six of 50 schools had pre-existing relationships 
with providers. Schools that were developing new partnerships or had new principals (principal 
turnover was reported in seven sites) reported greater challenges with developing principal support. 

A majority of providers utilized overlapping or blended staffing models, which helped to 
ensure continuity between the school day and OST program. Providers with these staffing 
models tended to have more consistently positive school- partnerships. Figure 8 describes the multiple 
staffing models that providers used to develop strong relationships with the host schools and describes 
the number of providers using these models. 
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Figure 8. Provider-School Partnerships: 21 st CCLC Provider Staffing Models 


Blended Staff. Teachers or paraprofessionals from the 
school day work in the after-school program. 


No overlap or blended staffing. 


One provider hired only classroom teachers. 
One provider primarily hired 
paraprofessionals from the school. 

One provider hired some classroom 
teachers and paraprofessionals from 
the school to staff the program in at 
least half of their sites. 


Staff overlap. OST site coordinator 
hours overlap with school day, allowing 
them to be present in school building 
during the day. 

• Four providers had overlapping hours for 
site coordinators and other staff who were 
present in the building during the school day. 



Three providers had neither blended staff nor 
overlap with school hours for their staffing. 


Staffing Model 
Frequency 



Blended and 
overlapping staffing. 

• One provider hired some teachers 
and paraprofessionals to work in 
their program AND had overlapping 
work hours. 


Areas for Growth: Robust School Partnerships 

Our analyses identified a number of areas that are still challenging for providers and present 
opportunities for further development of strong school partnerships. 

A few providers experienced challenges establishing trust and mutual respect with new 
principals. In only a handful of cases, providers that were developing relationships with new schools 
or new principals reported challenges. For example: 

• Several sites with overlapping and blended st affin g models experienced challenges with 
clarifying the role of staff that were present in the school all day so that they did not get tasked 
with inappropriate duties, such as security. 

• Other sites experienced challenges with clarifyingthe principal's authority over OST staff and 
programmatic decisions. In one case, one provider with a blended staffing model encountered a 
principal who felt uncomfortable with her staff taking direction from the OST provider for after- 
school activities. Another provider discovered that the principal expected them to get 
permission from her for taking youth on after- school trips. 

• Providers also reported challenges in getting access to some principals to communicate about 
program needs or challenges. 

These types of challenges were mostly encountered in new OST-prindpal relationships. 

Low morale existed in two sites where schools were slated for closure. Not surprisingly, 
providers had difficulty gaining traction for the program and recruiting and retaining participants in 
programs at these schools. Both programs ended early in the school year. 


Providers were less likely to have multi-leveled partnerships with schools, particularly 
relationships with multiple teachers. Providers varied significantly in terms of the depth of the 
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partnerships they developed with schools. This was particularly true regarding the degree to which they 
developed relationships with the classroom teachers of students in their programs. This variation 
included the following: 

• Six of the ten providers who completed the staff survey reported contact with only one or two 
teachers in each of their sites. 

• Blended and overlapping staff models generally supported more frequent and widespread 
contact with teachers but there were instances where OST staff were in the school building 
during the school day and did not necessarily use this time to build relationships with teachers. 

• Tensions were also reported with teachers because providers often had to share their space. 

• Our analyses identified only one of eleven providers that had site coordinators participating in 
school committees, which is another indicator of a multi-leveled relationship between the OST 
program and the school. 

Almost half of the providers did not have regular and reciprocal data sharing with 
schools. Providers were less likely to report having regular access to student data, such as report cards 
or school attendance, to inform program planning. Even among organizations that reported some 
access to student data, it was not clear that regular access was granted (e.g. daily school attendance or 
quarterly report card grades) . In addition, in some instances, data access was a result of students 
voluntarily sharing report cards, a strategy that is unlikely to provide complete data for all students in 
the program. Ideally, data sharing should be reciprocal— providers have program data, such as 
enrollment and participation data, which could be of interest to schools. It was not clear that many 
providers regularly shared this type of information with the schools. 

Alignment of program norms to school day norms was challenging for some sites. Our 

data also captured a handful of instances in which principals had concerns about how OST staff were 
managing student behavior in the program. This challenge may relate to the lack of alignment between 
program norms and school day norms. As discussed earlier, staff reported that managing student 
behavior was one of their greatest challenges. Research suggests that providers and schools need to 
agree on behavioral norms for the after- school program in order to strengthen their partnership. 

D. Section Summary 

The primary goals of 21 st CCLC programs are to improve academic achievement, student behavior, and 
social skills. Programs worked towards these goals through activities such as homework help and 
project- based learning (PBL). Programs that incorporated STEM components had goals aligned with 
indicators of high quality STEM programming: increasing student interest, career awareness, and 
content knowledge in STEM. Moreover, programs supplemented academic support with enrichment 
and skill- building activities to engage students. However, academic supports provided by programs 
were hampered by providers' limited capacity to provide one-on-one or small group academic support 
and the inconsistent alignment of programs with school curricula. In addition, STEM programming did 
not always include hands-on learning opportunities nor did staff across all programs receive the 
program planning and design support that they needed. 

This year's staff survey allowed RFA to examine staffing issues from the viewpoint of staff and not just 
through secondary information gathered from program directors and site coordinators. Overall, OST 
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staff members had appropriate educational credentials, but did not have a lot of experience with the 
providers or in the OST field. However, providers invested in strong leadership by hiring program 
directors and, in most cases, site coordinators. Moreover, staff had many opportunities to participate in 
professional development but identified several key areas where they felt they needed additional 
training, such as managing student behaviors. 

Finally, providers in our sample generally reported adeguate support from schools and principals. 
Providers encountered generally supportive principals who were grateful for the academic enrichment, 
and after-care support provided by the programs and saw these programs as valuable resources which 
supported and shared their vision and goals. More than half of providers had a blended or overlapping 
staffing model, which allowed them to develop partnerships with teachers and become more familiar 
with and aligned with the school curriculum. However, providers continued to experience some 
challenges in their partnerships with schools, such as aligning program behavioral norms with school- 
day norms and regularly accessing and sharing data with schools. 
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IV. Student Outcomes 

Research on out- of- school time (OST) programs suggests that these programs can have an impact on 
school attendance, academic outcomes - including standardized test scores, and sodo- emotional 
outcomes. 44 RFA conducted a series of analyses to evaluate the potential impact that the 21 st CCLC 
programs had on each of these outcome areas for elementary, middle, and high school students. In this 
section, we report the findings from our analyses of the following outcome areas: 

• School attendance 

• Reading, math and sdence course grades 

• Reading and math PSSA scores 

• 1 st - 3 rd grade Developmental Reading Assessment (DRA) 

• High school student credit accumulation 

• Student behavior 

A. Data and Analysis 

The analyses of student outcomes use a subset of public school students that attended the 21 st CCLC 
programs to examine the relationship between student outcomes and program partidpation. The subset 
of programs used for analyses met the following criteria: 

• Completion of a full year of programming in 20 12- 13 

• Availability of two years of data for the analysis 

• Availability of partidpation data 

• Utilization of PSSA assessments 

• Use of common grading systems 

A total of 40 sites, representing ll 45 providers, met these criteria. 46 Data on program content areas and 
program staff were colleded via the staff survey described in the previous sedion. Our analyses also 
compare OST regular partidpants with their peers, including students who attended the same schools 
but did not partidpate in 21 st CCLC programs. Grade levels and assessment data for all OST partidpants 
and non- OST students were provided by the School Distrid of Philadelphia, and one charter school. 

RFA utilized the following definitions of OST groups for the evaluation: 

Regular Participants: Partidpants who attended at least 30 days of programming. The Regular 
Partidpant group is further broken down into 30-89 day partidpants and 90 + day partidpants in our 
descriptive analyses. 


44 Durlack, R & Weissberg, R ( 20 12) . After- school programs that follow evidence- based practices to promote sodal and emotional 
development are effective. Expanded Learning and After-school: Opportunities for Student Success. 

http:// www.expandindeaminq.org/ docs/ Durlak&Weissberg FinaLpdf : Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School 
Programs in the 21 st Century; Their Potential and What it takes to Achieve it. Issues and Opportunities in Out of School-Time Evaluation: Issue 
10. Cambridge, MA: Harvard Family Research Project 

45 One provider did not participate in stuff survey so the total provider is 10 for analysis using stuff survey data. 

46 Eight Catholic schools could not be incorporated because they utilized Terra Nova assessments rather than the PSSA exams. In addition, the 
structure of grading systems in Catholic schools differed from the public schools making it difficult to merge the data. Two public sites did not 
provide participation data. 
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• 30-89 day Participants: Participants who attended 30-89 days 

• 90 + day Participants: Participants who attended 90 or more days 

A Comparison Group was selected from the less than 30 day participants or Non-OST students as 
defined below: 47 

• Less than 30 day Participants: Participants who attended fewer than 30 days 

• Non-OST Students: Students who did not participate in the program 

Limitations of the Study 

Although RFA's analyses took into account the observed differences in student background 
characteristics, as well as the differences in student academic and behavioral performance before 
participating in the 2 1 st CCLC program in 20 12- 13, selection bias remains a possibility, given the limited 
student and school information available for this study. Furthermore, we did not have data on whether 
OST participants and non- OST students attended programs other than 2 1 st CCLC programs. As a result, 
the analyses could be biased if some comparison students included in this study attended other 
enrichment programs. 

Organization of Findings on Student Outcomes 

The analyses in each outcome area are organized in the following way: 

1. Descriptive analyses of OST participants' and non-OST students' outcomes. The 

descriptive analyses provide a snapshot of the student academic and behavioral outcomes by 
OST participation levels. However, OST participants and non-OST students differ by their 
demographic and family characteristics, such as race/ ethnicity, socioeconomic status, gender, 
English proficiency levels, disability status, as well as by their historical academic achievements. 
The observed differences presented in the descriptive analysis may be due to these pre-existing 
differences and may not reflect the potential impact of the 21 st CCLC OST programs. 

2. Comparative analysis between OST regular participants and a comparison group 
that includes non-OST students and participants who attended 21 st CCLC programs 
less than 30 days. Analyses compare regular participants with a selected comparison group of 
students with similar background characteristics and academic achievement before attending 
the OST programs in 20 12- 13. These analyses are intended to determine whether OST regular 
participants outperformed comparison students if there was no preexisting statistically 
sig nifi cant difference in student achievement and observed background characteristics based on 
the available data. 

3. The relationship between student outcomes and varying participation levels of the 
21 st CCLC programs participants. RFA conducted a set of analyses using program 
participation data to examine the relationship between OST days attended and student 
attendance, taking into account the differences in race/ ethnicity, gender, English proficiency 
levels. Free Reduced Lunch (FRL), and disability status. 


47 RFA utilized the Propensity Score Matching method (nearest naghbor matching within a caliper) to select a comparison group for the 
comparative analysis. Student demographic characteristics, as well as students' prior academic achievement and behavioral performance were 
controlled in the regression analyses. 
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We also incorporated the organizational- level elements of program quality, such as staff characteristics, 
content focus of the OST program, and strength of the school- program partnership into our analyses, 
which allows us to examine the role of these provider- level factors in the observed student outcomes. 
We summarize the finding from these analyses after reporting the results for all the outcome areas. 

B. Student Attendance 

The analysis of student attendance examines the likelihood of OST regular participants having 10 or 
more unexcused absences in comparison to participants who attended less than 30 days and non- OST 
students, and in relationship to their level of program participation. This section of the report begins 
with a descriptive analysis of the school attendance data of OST subgroups and non- OST students, and 
then provides results from the comparative analysis between OST regular participants and the 
comparison students with similar background characteristics and prior academic and behavioral 
performance. We then report the relationship between participation levels and student school 
attendance. 

Descriptive Analyses 

Figure 9 illustrates the percentage of students who had 10 or more unexcused absences in 20 12- 13 by 
grade and participation level. The dashed line separates the groups who regularly participated in the 
programs from OST participants who attended less than 30 days of the programming and non- OST 
students. 


Figure 9. Percentage of Students Having 10 or More Unexcused Absences by Grade and Participation Level 



• With the exception of middle school students who attended 30-89 days of programming, across 
all the grade levels, a smaller percentage of regular participants had 10 or more unexcused 
absences in comparison to less than 30 day participants and non- OST students. 

• Across all OST groups, the percentage of students having 10 or more unexcused absences was 
lower for students in lower grade levels. 
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Comparative Analyses 

After ruling out differences in student background characteristics and student performance existing 
before entrance in the OST programs, our analyses suggest the following: 48 

• Elementary school OST regular participants were less likely to be absent 10 or more days than 
comparison students. 

• For middle and high school students, there was no difference in the likelihood of being absent 
for 10 or more school days between OST regular participants and comparison students. 

Relationship Between Level of Program Participation and School Attendance 

Figure 10 displays the results of the logistic regression analysis showing changes in school attendance as 
program participation increases, controlling for the observed student and site characteristics, as well 
preexisting differences in student academic and behavioral performance. In Figure 10, the three lines 
show the estimated probabilities of having 10 or more unexcused absences at each participation level 
and the vertical line marks 30 days of participation in the OST program. For example, the probability of 
the having 10 or more unexcused absences for high school students (orange line) who did not 
participate in OST programs was approximately 0.8 (or eight out of 10). In contrast, high school 
students who attended 120 days had a much lower probability of 10 or more absences (roughly 0.3 or 
three out of 10). 


Figure 10. Relationship Between OST Program Participation and School Attendance 49 


Q) 

O 

c 

0 

cn 

_Q 

0 

"O 

0 

0 


1 - 
.9- 
. 8 - 
.7- 


.5- 


U - 6 
— 0 

S § 

3D 0 

2 O 4- 
Q- E 

o .3- 

0 

o> .2- 
_c 

1 .1 - 


0- 



1 30 60 90 120 150 180 

Days Attended in OST Program 


Elementary Middle High 


• For all grade levels, students who attended more days in the OST program were less likely to 
have 10 or more unexcused absences than students who participated fewer days. 50 


48 Detailed results from the comparative analyses are provided in Table B2 in Appendix B. 

49 p<0.05 for the relationship displayed in the figure. All other variable considered in the analyses are held constant. Detailed results are 
reported in Table Cl in Appendix C. 

50 For example, elementary students who attended 30 days in the OST program have 28% lower odds to have 10 or more unexcused absences 
than those who attended only one day. 
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C. Reading and Math Course Grades 

The analysis of literacy and math outcomes examines students' reading and math courses grades 
relative to a comparison group, and in relationship to their level of program participation. Like in the 
previous section, we first present a descriptive analysis of the course grade data of OST subgroups and 
non- OST students, followed by the results of comparative analyses between OST regular participants 
and the comparison groups with similar background characteristics and prior academic and behavioral 
performance. We end the section with an analysis of the relationship between participation levels and 
student course grades in reading and math. 

Descriptive Analyses 

Figure 11 illustrates percentage of student earning a "C" or above by grade and participation level. The 
dashed line separates the groups that regularly participated in the programs from OST participants 
attending less than 30 days and non- OST students. 


Figure I I . Course Grades: Percentage of Students Earning a C or Above, by Grade and OST Subgroup 


Elementary (1st-5th) Middle (6th-8th) 




Reading 


Math 


Science 


• Math: A higher percentage of 90 + day participants in elementary schools earned a C or better 
in math as compared to participants attending less freguently and non- OST students. 

• Science: Overall, a very large percentage (greater than 95%) of elementary students, including 
all OST participant groups and non- OST students, received a "C" or better in science. 

• All Subjects: A higher percentage of 90 + day participants at the middle school level earned a C 
or better in all subject areas in comparison to participants attending less freguently and non- 
OST students. 
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Comparative Analyses 

After taking into account any pre-existing differences in student background characteristics and student 
performance before entering into the OST programs, our analyses suggest the following: 51 

• Regular participants in middle school, on average, earned higher scores in math and reading 
than comparison students. 

• There is no significant difference in middle school science course grades between regular 
participants and comparison students. 

• There is no significant difference in elementary school reading, math, and science course grades 
between OST regular participants and comparison students. 

Relationship Between Level of Program Participation and Student Course Grades 
Figures 12- 15 further illustrates the results from the regression analyses showing changes in student 
course grades as OST program participation increased, controlling for all other student and site level 
factors. The area within the dashed lines defines the confidence interval, or range, in which scores could 
fall. As the range gets wider, uncertainty increases. The vertical line marks 30 days of participation in 
the OST program. 


51 Detailed results of the comparative analysis controlling for various student characteristics are provided in Table Blin Appendix B. 
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Figures 12-15. Relationship Between Level of OST Participation and Elementary School Students Course Grades 52 


Reading Course Grade(1-5) 



Math Course Grade(6-8) 



Reading Course Grade(6-8) 



Science Course Grade(6-8) 



52 p<0.05 for the relationship displayed in the figure. All other variable considered in the analyses are held constant. Detailed results are reported in Table C2 in Appendix C. 


• On average, elementary school OST participants who attended more program days received 
higher course grades in reading than participants who attended fewer days. 

• On average, middle school OST participants who attended more program days received higher 
course grades in reading, math, and science than participants who attended fewer days. 

• No relationship was found between levels of program participation and elementary course 
grades in math and science. 


D. Reading and Math PSSA Scores 

The analysis of student performance in reading and math PSSAs compares OST participants' scores on 
reading and math PSSAs to that of students in the comparison group, and in relationship to their level 
of program participation. The section begins with a descriptive analysis of the PSSA score data of OST 
subgroups and non- OST students, and is followed by the results from the comparative analysis between 
OST regular participants and the comparison students with similar background characteristics and 
baseline academic and behavioral performance in the previous year. The section ends with the analysis 
examining the relationship between participation levels and student PSSA scores in reading and math. 


Descriptive Analyses 

Figure 16 illustrates the percentage of students who scored proficient or advanced on the PSSA exam in 
20 12- 13 by grade and participation levels. The dashed line separates OST regular participants from 
participants who attended less than 30 days, and non- OST students. 

Figure 1 6. PSSA: Percentage of Students Scoring Proficient or Above by Grade and OST Subgroup 


Elementary (4th-5th) Middle (6th-8th) 



• Higher percentages of 90 + day OST participants scored proficient or advanced on the math and 
reading PSSA than participants who attended fewer days of OST programming or non- OST 
students. 
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Comparative Analyses 

After ruling out differences in student background characteristics and student performance that existed 
before entrance in the OST programs, our analyses suggest the following 53 : 

• For both elementary and middle school students, there is no significant difference on reading 
and math PSSA between regular participants and comparison students. 

Relationship Between Level of Program Participation and Student Performance on PSSA 
Figure 17- 19 illustrates the results of a regression analysis showing changes in student math and 
reading PSSA scores as OST program participation increased, after controlling for background 
characteristics. Again, the vertical line (x=30) displays 30 days participation in the OST program. The 
area within the dashed lines defines the range within which we can be 95% certain the predicted values 
will fall. As the area between the dashed lines gets wider, the uncertainty increases. 

• On average, elementary school students with higher OST attendance scored higher on both math 
and reading PSSA than OST participants who attended fewer days. 

• Middle school students with higher OST attendance scored higher on the reading PSSA than 
OST participants who attended fewer days. 


53 Detailed results are reported in Table B1 in Appendix B. 


31 


Figure 17-19. Relationship Between OST Participation and Student Performance on PSSA 54 


PSSA Reading (4-5) 



PSSA Reading(6-8) 



PSSA Math(4-5) 



54 p<0.05 for the relationship displayed in the figure. All other variable considered in the analyses are held constant. Detailed results are reported in Table Clin Appendix C. 
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E. Developmental Reading Assessment 

The Developmental Reading Assessment (DRA) is included in our analysis as an outcome for 1 st - 3 rd 
grade students who do not take the PSSA or do not yet have two years of PSSA data available. The 
analysis of DRA examines OST regular participants' reading assessment scores in comparison to those 
that participated less than 30 days and non- OST students, and in relationship to their level of program 
participation. This section of the report starts with a descriptive analysis of the developmental reading 
assessment data of OST subgroups and non- OST students, and is followed by the results from 
comparative analyses between OST regular participants and the comparison group with similar 
background characteristics and prior academic and behavioral performance. We then report on the 
analysis examining the relationship between participation levels and DRA scores. 

Descriptive Analyses 

Figure 20 illustrates the percentage of 1 st through 3 rd grade students reading at grade level by grade and 
participation level. 

Figure 20. Percentage of Students Reading at Grade Level, by Grade and OST Subgroup 



• Higher percentages of 90 + day participants read on grade level than participants who attended 
fewer than 30 days in OST programming and non- OST students. 

Comparative Analyses 

After taking into account differences in student background characteristics and student performance 
that existed before entrance in the OST programs in 20 12- 13, our analyses suggest the following 55 : 


• Regular participants and comparison students displayed egual likelihoods of reading at grade 
level. 


55 Detailed results are reported in Table B2 in Appendix B. 
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Relationship Between Level of Program Participation and Student Performance on DRA 
Figure 21 displays the results of the logistic regression analysis examining the probability of reading on 
grade level as program participation increases, controlling for the observed student and site 
characteristics, as well as preexisting differences in student academic and behavioral performance. The 
area within the dashed lines defines the range within which we can be 95% certain the predicted values 
will fall. As the area between the dashed lines gets wider, uncertainty increases. The vertical line marks 
30 days participation in the OST programs. 

Figure 21. Relationship Between OST Program Participation and Student Performance on DRA 56 



• Students who attended more days of OST programming were more likely to read on grade level 
than students who participated fewer days. 57 

F. High School Student Credit Accumulation 

This section presents analyses of high school student credit accumulation in English/ Language Arts 
(ELA), mathematics, and science comparing OST regular participants to participants who attended less 
than 30 days and non- OST students, and in relationship to their level of program participation. We 
examined high school credit accumulation because not all high school students in our study take 
standardized tests. Although 11 th grade students take the PSSA there were not two consecutive years of 
standardized test scores available for our analyses. In addition, the complexity of high school course- 
taking patterns makes it challenging to aggregate and compare course grades. 

Descriptive Analyses 

Figure 22 illustrates percentage of students who earned all the credits attempted in each subject area of 
ELA, math, and science in 20 12- 13. 


56 p<0.05 for the relationship displayed in the figure. All other variable considered in the analyses are held constant. Detailed results are 
reported in Table Cl in Appendix C. 

57 For example, students who attended 30 days in the OST program have 19% higher odd of reading on grade level than those who attended 
only one day. 
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Figure 22. Percentage of high school students earning the credits attempted by subject and OST subgroups 



• Overall, a higher percentage of OST participants earned credits attempted in all subj ect areas 
than non- OST students; and 

• Across all subject areas, a strong maj ority of all students received all credits attempted. 
Comparative Analyses 

The comparative analyses, controlling for preexisting differences in academic and behavioral 
performance and observed background characteristics, suggest the following: 58 

• For each of the subject areas studied, the probability of earning all credits attempted is the same 
for OST regular participants and comparison students. 

Relationship Between Level of Program Participation and High School Student Credit Accumulation 
Figure 23 displays the results from logistic regression analyses showing changes in the likelihood of 
earning all the credits attempted in ELA and math as OST program participation increases, controlling 
for all other student and site level factors. The vertical line marks 30 days of participation in the OST 
program. The area within the dashed lines defines the range within which we can be 95% certain the 
predicted values will fall. As the area between the dashed lines gets wider, uncertainty increases. The 
vertical line marks 30 days participation in the OST programs. 


58 Detailed results are reported in Table B2 in Appendix B. 
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Figure 23. Relationship Between OST Program Participation and High School Student Credit Accumulation 59 
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• Students who attended more days in OST programming were more likely to successfully earn all 
credits attempted in ELA and math than those who attended fewer days. 60 

G. Student Behavior Outcomes 

RFAs analyses of student behavior outcomes examined the likelihood that OST regular participants 
would receive out- of- school suspensions in comparison to their peers, and in relationship to their level 
of program participation. 

Descriptive Analyses 

Figure 24 illustrates the percentage of students who had one or more out- of- school suspensions in 
20 12- 13 by grade levels and OST participation groups. The dashed line separates the groups who 
regularly participated in the programs from other groups such as OST participants who attended less 
than 30 days of the program or non- OST students. 


59 p<0.05 for the relationship displayed in the figure. All other variable considered in the analyses are held constant. Detailed results are 
reported in Table C2 in Appendix C. 

60 For example, students who attended 30 days in the OST program have 36% higher odds to earn all credits attempted in math than those who 
attended only one day. 
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• A smaller percentage of 90 + day OST participants had an out- of- school suspension in 20 12- 13 
than participants who attended fewer days or non- OST students. 

Comparative Analyses 

When comparing OST regular participants with students with similar background characteristics and 
baseline performance in the previous year, our analyses suggest that: 61 

• OST regular participants and comparison students had the same likelihood of having one or 
more out- of- school suspensions. 

Relationship Between Level of Program Participation and Student Suspensions 

Figure 25 displays the results of a regression analysis showing changes in the likelihood of receiving one 
or more out- of- school suspensions as program participation increased. The vertical line (x=30) displays 
30 days participation in the OST program. The area within the dashed lines defines the range within 
which we can be 95% certain the predicted values will fall. As the area between the dashed lines gets 
wider, uncertainty increases. The vertical line marks 30 days participation in the OST programs. 


61 See detailed results in Table B2 in Appendix B. 
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Figure 25. Relationship Between OST Attendance and Student Suspensions 6 
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Students who participate in OST programs more frequently are less likely to receive one or more 
suspensions than those participating less frequently. 63 


H. The Role of Organization-Level Elements of Quality Programming 

21 st CCLC providers vary by program goals and content, staff characteristics, school partnerships and in 
other observed and unobserved ways. The degree to which organization- level elements of quality 
programming are related to program effectiveness may have policy and practice implications for 
providers and other education and OST policymakers. RFA's evaluation incorporated data collected 
from a staff survey which assessed these organization- level elements of quality identified in other 
research as supports for high quality OST programs into our analyses. This section provides a summary 
of the findings of these statistical analyses. More details about the analyses and the results are reported 
in Tables C1-C4 in Appendix C. 64 


• Staff with college degrees: A higher percentage of staff with at least a bachelor's degree is 
associated with a higher probability that OST high school students will successfully earn all 
credits attempted in each subject area of Reading, Math, and ELA. However, the relationship 
between the percentage of staff with a bachelor's or higher degrees and student performance is 
mixed across different assessments or subject areas for elementary and middle students. There 
is also no consistent relationship pattern between staff degree and behavioral outcomes of all 
the students. 


62 p<0.05 for the relationship displayed in the figure. All other variable considered in the analyses are held constant. Detailed results are 
reported in the appendix Table C4. 

63 For example, dementaiy students who attended 30 days in the OST program have 22% lower odds to have out- of- school suspension than 
those who attended only one day. 

64 The variables actually included in the analyses vary for different analyses because of missing data and oolinearity issues, or due to little 
variation across providers within the grade level studied. 
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• Staff with teaching certifications: A higher percentage of staff with a teacher certification is 
associated with a higher probability that OST high school students will successfully earn all 
credits attempted in the subject areas of ELA and math. However, there are no consistent 
relationship patterns between the percentage of staff with a teacher certification and other 
outcome areas studied. 

• OST staff meetings with school teachers: A higher percentage of staff meeting with 
teachers is negatively associated with elementary and middle school course grades as well as 
elementary students' school attendance and behavioral outcomes. However, this is positively 
associated with reading PSSA scores for both elementary and middle school students. 

• OST staff meetings with principals: A higher percentage of staff meeting with principals is 
negatively associated with elementary and middle school reading PSSA scores, elementary 
reading course grades, as well as elementary and middle school attendance. 

• Other organization-level elements of guality programming: The analyses of other 
organizational- level elements, such as percentage of full-time staff, percentage of staff majoring 
in STEM, percentage of staff using curriculum for planning OST activities, percentage of staff 
considering a strong academic or STEM as a program, etc., also produced mixed results. 

Overall, our analyses found much variation across the outcome areas of the study (attendance, course 
grades, PSSAs, behavior, and others) and grade levels. The mixed results from the analyses reflect 
complex relationships between organization- level elements of guality programming and student 
outcomes. More specifically, while the organization- level elements of guality programming can play an 
important role in achieving the proposed academic and behavioral goals of 21 st CCLC OST programs, 
the presence of these elements among providers might also be a function of the needs of students 
served. For example, on one hand, the positive relationship between staff guality in terms of percentage 
of staff with at least a bachelor's degree or with a teacher cer tifi cation and high school student credit 
accumulation might suggest the importance of having gualified staff for the programs serving high 
school students, which is consistent with the findings documented in the literature. On the other hand, 
the negative relationship between the percentage of staff meeting with school teachers or principals and 
student outcomes, such as course grades and school attendance, might reflect a stronger partnership 
between OST programs and schools as a response to the needs of low-performing students. 

These findings should be interpreted with caution given the limitations of the analyses and 
complex relationships between organization- level elements of guality programming and measured 
student outcomes for all outcome areas and grade levels. 

I. Section Summary 

REA analyses examined the potential impact of OST programs on student academic performance and 
behavioral outcomes. After taking into account the pre-existing differences between OST and non- OST 
students, 21 st CCLC OST regular participants did better than students who attended less than 30 days in 
OST programs and non- OST students in multiple arenas. Specifically: 

• Regular elementary school OST participants were less likely than comparison students to have 
10 or more unexcused absences. 

• Regular middle school OST participants were more likely than comparison students to earn 
higher course grades in math and reading. 
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For OST participants, higher levels of participation were associated with better outcomes in 
many, but not all, behavioral and academic performance indicators. Table 4 summarizes 
these participation- level findings. 

Table 4. A Summary of Finding from Analyses of the Relationship Between OST Participation Levels and Student Outcomes 


Outcomes 

Elementary 

Middle 

High 

Fewer school absences 

• 

• 

• 

Higher reading course grades 

• 

• 

• 

Higher math course grades 

• 

• 

• 

Higher science course grades 

• 

• 

• 

Score higher on PSSA reading 

• 

• 

• 

Score higher on PSSA math 

• 

• 

• 

l st -3 rd grade students reading at grade 

• 

• 

• 

Successfully earning credit attempted 
in ELA and math 

• 

• 

• 

Successfully earning credit attempted 
in science 

• 

• 

• 

Fewer out-of-school suspensions 

• 

• 

• 


Significant evidence 


No significant evidence 


N/A 
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V. Recommendations 

The results of this cross- site analysis of OST programming and participation outcomes offer insights 
into the strengths and challenges of OST programs in Philadelphia and suggest recommendations for 
providers to strengthen their individual programs and for policy-makers to continue building the 
capacity of the City's OST system. 

A. Recommendations for Program Providers 

• Focus on Retention. Our study suggests that when students participate more freguently in 
OST programs, they have better academic and behavioral outcomes. This confirms what has 
been reported in other research on OST programs 65 . Rather than expanding enrollment, 
programs could work to increase participation and retention among those already participating, 
particularly for middle and high school students. Intentional retention efforts, such as 
monitoring program attendance and calling students who drop off from the program and their 
parent/ guardians may be reguired with older youth. 

• Increase the percentage of program staff with bachelor's degrees and explore ways 
to increase the percentage of certified teachers for programs serving high school 
students. Hiring more cert ifi ed teachers and staff with bachelor's degrees is associated with 
some positive academic outcomes for high school students. This co nfir ms what has been 
reported in other literature on OST programs 66 . However, hiring staff with more education and 
teacher cer tifi cations can be costly. 21 st CCLC OST programs should consider ways to hire at 
least one certified teacher to oversee and mentor other front-line staff in providing academic 
support. 

• Providers should continue to develop relationships with school personnel, 
particularly teachers. Providers in our study varied in the extent to which they developed 
relationships with the classroom teachers of students they served. Relationships with classroom 
teachers allow for more continuity between school day activities and norms and after- school 
activities. Schools and OST providers could work together to identify the best strategies for 
developing these relationships. Strategies may include periodic common planning time for 
teachers and OST staff, or communications systems such as "homework logs" 67 , through which 
teachers could share information about daily homework. OST staff and schools could also 
explore common professional development that would promote alignment of supports for 
students. 

• Consider blended or overlapping staffing models and leverage their potential. This 
study, as well as other evaluations of OST programs, suggests that blended staffing or 
overlapping st affin g models in which OST staff work either for the school or in the school during 
the school day, may be a promising practice for building school partnerships. This blended staff 
model can provide additional opportunities for OST staff to build relationships with the school 


65 Deschenes, S. N., Arbreton, A., Little, P.M., Herrera, C., Grossman, J.B., Weiss, H.B., Lee, D. (2010). Engaging older youth: Program and 
city- level strategies to support sustained participation in out-of-school time. Harvard Family Research Project. 

66 Bodilly, Susan J . and Megan K. Beckett. Making Out- of- School-Time Matter: Evidenoe for an Action Agenda. Santa Monica, CA: RAND 
Corporation, 2005. http://www.rand.org/ pubs/ monographs/ MG242 . Also available in print form. 

67 Huang, D., Dietel, R. (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California 
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and ensure the alignment of the school day to the OST program. For example, providers with 
staff presence in the school were able to meet with classroom teachers, participate in school 
committees or teacher professional development sessions, and generally stay abreast of the 
major events or developments occurring during the school day. In our study, these models 
support but do not guarantee the development of these deeper school partnerships. In some 
cases, school- based site coordinators did not fully leverage their presence in the school to 
deepen the school partnership. Providers should be intentional, then, about utilizing staff based 
on schools to develop school relationships. 

• Enhance STEM. Strong STEM programming is characterized by hands- on, inguiry- driven 
learning opportunities which encourage a high level of activity, engagement and STEM 
knowledge. In this first year of widespread STEM programming, RFA's observations suggest 
that there is room for growth in developing STEM programming across providers; particularly 
when OST staff are asked to deliver the STEM curriculum. OST staff should have a clear 
understanding of the material but also the ability to facilitate an inguiry driven, hands-on 
program, in order to positively influence student interest in STEM, their awareness of STEM 
career opportunities, and eventually their success in STEM courses. 

• Consider ways to provide more subject-specific and one-to-one tutoring efforts. 

Programs can have the greatest impact on academic outcomes when they provide regular one- 
to-one tutoring supports. Programs should consider strategies, such as recruiting high school or 
college student volunteers, to provide these supports. However, this would require programs to 
create an infrastructure to support tutors, including appropriate supervision and training. 

• Engage in reciprocal data sharing. Strong school partnerships are fostered by reciprocal 
data sharing. Sharing student grades, test scores, school attendance and suspensions data can 
help OST providers better target their individual or collective support for participants. As 
providers seek access to data from schools, they should also look for opportunities to share data 
on their programs with schools including enrollment and participation data or local evaluation 
reports. School staff could benefit from knowing which students are participating in the after- 
school program so that they can leverage their supports. 

• Leverage 2 1 st CCLC advisory councils to build staff competencies. Across providers, 
staff reported that building relationships with parents and utilizing community resources were 
their most challenging tasks. Research on OST programs suggests that engaging parents and 
communities is a best practice which increases the engagement and impact of OST programs. All 
21 st CCLC programs are required to have advisory councils made up of parents and other 
stakeholders. Programs should leverage these groups to help build staff skills in this important 
area. 

B. Recommendations for Philadelphia’s OST System 

Continue to Support Citywide Professional Development of OST Staff in the following areas: 

• Train the trainer professional development for providers on supporting their 
front-line staff with behavior management: Staff reported behavior management as 
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one of their most challenging areas. In addition, staff, across providers, tended to be newer 
to the OST field and may have lacked experience in this area. Given that staff turnover is 
often high in the OST field, supporting staff with behavior management is an ongoing need 
for OST organizations. Philadelphia's OST system already offers professional development 
opportunities for OST staff around behavior management. The system could further support 
providers by providing them resources to offering trainings for program directors and 
managers regarding best practices and resources for coaching OST staff in this area. 

• Building community connections and relationships with parents: Philadelphia's 
OST system could also support providers in identifying ways to build skills and knowledge 
related to their neighborhoods, community resources and parents. 

• Implementation of STEM programming: Philadelphia's OST system has made 
available the resources of the Franklin Institute to help train OST providers in offering 
STEM programming. However, our observations as well as student feedback suggest that 
there is room for growth in this area. Research indicates that high quality STEM programs 
are characterized by opportunities for students to engage in hands-on, inquiry- based 
learning environments. To increase the quality of STEM programming in the OST system, 
additional professional development and coaching, particularly on the hands-on and 
inquiry- based approach may be needed to develop OST staff capacity in this area. 

• Increasing student participation in programs, particularly engaging middle and 
high school students: Middle school and high school students had low levels of 
participation in the 21 st CCLC OST programs and program participation was related to 
improved outcomes for students. Programs could benefit from support in thinking about 
how to better engage middle and high school students so that they can retain more youth in 
the programs, support youth skill development and thus, improve student outcomes. 

• School curriculum and state standards: Even if academic supports are not central to a 
providers' program model, homework help sessions could improve if program staff were 
familiar with the topics that students are expected to learn. For older students, it would be 
beneficial for staff to learn more about Keystone exit exams and consider ways to support 
older students in passing those exams. 


Work with the School District of Philadelphia to Develop School-Level Support for OST-School 
Partnerships. The District could help strengthen OST programs in the following ways: 

• Increase principal awareness of the value of OST programs and ways in which they 
can support these programs in their schools. Principal support is essential when 
integrating the OST program in the life of the school and developing alignment between the 
school day and the OST program. Principals who are invested in OST programs in their schools 
could be invited to speak to other principals at a forum either hosted by the School District or 
another organization. It would also be important to provide encouragement and support for 
principals about ways to deepen the partnerships between OST and school staff and create 
opportunities for further communication between OST staff and teachers. 
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• Encourage schools to have common professional development with their OST 
providers about the curriculum and academic improvement needs of their 
students. Common professional development is another strategy for aligning OST and school 
day activities and facilitating the development of relationships between OST staff and classroom 
teachers. School District staff could provide professional development to OST providers about 
the District curriculum. OST providers could provide training for teachers on project- based 
learning or STEM programming. 

• Make student data accessible to program providers in real time so that they can 
target academic interventions for program participants. OST programs could better 
target academic and behavioral interventions if they had real-time access to student 
benchmarks, report cards, daily attendance and disdplinaiy incidents. 

Consider Ways in Which the City Could Support the Development of the OST Workforce. 

Clearly this is one of the most significant issues facing the city OST programs. There are no easy 
answers but system- level OST decision-makers should continue to consider strategies for creating a 
larger pool of highly gualified staff for OST programs such as increasing salaries and offering training 
opportunities that are substantial and accessible . 68 

C. Recommendations for Future Research 

The results of this study encourage us to consider how we can build upon this work to offer further 
insight into factors supporting the development of high-quality OST programs and promising practices 
that support positive student outcomes. The following types of studies are worth considering in the 
future: 


• Longitudinal analyses. Studying the long term impact of OST programs on student academic 
achievement and other sodo- emotional outcomes would help fill a void in the OST research 
literature and provide the OST community with valuable information about the impad of 
sustained partidpation on student outcomes. 

• Best practices case studies. Best pradice research identifies programs that are proven 
effedive and examines in more depth their program and operational pradices. Research of this 
type would provide more information about strong organizational and program- level fadors 
which are effedive in Philadelphia. This type of investigation would employ a case study 
approach where intense observations, interviews, and other forms of qualitative data are 
colleded to understand program quality and best pradice implementation. 

• System-change study. As noted above, efforts are underway to support the development of a 
more cohesive OST system in Philadelphia. As these efforts progress, it will be important to 
understand how the system impads providers' ability to access data, resources, and other 
important information that will inform program improvement. 


68 Raley, R., Grossman, J., &Walker, K. E. (2005). Getting itright: Strategies for after- school success. Philadelphia, PA: Public/ Private 
Ventures. 
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Appendix A. Organization-Level Elements of Quality Programming Data 
Collection and Analysis 

Research for Action conducted one round of interviews with 21 st CCLC program directors, site 
coordinators, select school principals, and select student participants as well as administered a staff 
survey during spring 20 13. Researchers also collected and analyzed program documents, e.g.. Quarterly 
Performance Reports (QPRs) and program schedules. Table A1 belowprovides a brief description of the 
type of data collection tolls employed and the number of respondents. 

Table At Description of Data Collection 


Data Collection Tool 

Number of Respondents 

Staff Survey 

288 

Interviews with program directors and site coordinators 

131 

Interviews with principals 

16 

Interviews with students 

47 


Interviews were transcribed and data were coded using Adas. ti. Analytic memos were developed for 
each local provider covering several themes including program content, staffing, and school 
partnerships. Provider memos were the basis for the provider- level local evaluation reports. Data was 
then combined into three matrices (academic program content, staffing, and school partnerships) that 
reflected data across all 50 sites. Data from these matrices formed the basis for dty- wide analytic 
memos on the same three topics. 
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Appendix B. Comparative Analyses: Regular Participants Versus Comparison 
Students 

As discussed in the report, RFA conducted two sets of analyses to examine the potential impacts of OST 
programming on student academic performance and behavioral outcomes. The first set of analyses 
examine whether students regularly attending OST programs had better academic and behavioral 
outcomes than the comparison students. This part of the study uses propensity score matching methods 
to select a comparison group of students who did not regularly participate in the 2 1 st CCLC OST 
program in 20 13. Table B1 compares student academic performance on PSSA and course grades of 
regular participants and comparison students. 


Table Bl. PSSA and Course Grades: Regular Participants vs. Comparison Students 



Regular participants 

Comparison Students 

Difference 


n 

Mean 

n 

Mean 


Elementary 






PSSA Math (4-5th Grade) 

223 

1171.4 

227 

1207.1 

35.80 + 

PSSA Reading (4-5th Grade) 

224 

1114.0 

229 

1132.8 

18.80 

PSSA Science (4th Grade) 

129 

1211.0 

136 

1219.2 

8.20 

Reading Course Grades 

535 

80.8 

516 

80.9 

0.13 

Math Course Grades 

535 

81.1 

516 

81.4 

0.28 

Science Course Grades 

535 

84.1 

516 

84.4 

0.27 

Middle School (6-8th Grade) 






PSSA Math(6-8 th Grade) 

613 

1253.3 

544 

1234.1 

-19.18 

PSSA Reading(6-8 th Grade) 

614 

1198.1 

541 

1197.4 

-0.70 

PSSA Science (8 th Grade) 

146 

1076.4 

135 

1104.3 

27.90 

Reading Course Grades 

603 

78.9 

521 

77.8 

-1.10* 

Math Course Grades 

603 

78.1 

521 

76.6 

-1.44* 

Science Course Grades 

603 

79.6 

521 

79.6 

0.06 


tp<O.IO; *p<0.05; **£><0.01 


Table B2 reports estimated odds ratio from logistic regression analyses. The analyses examine: whether 
regular participants are less likely to have out- of- school suspensions or 10 or more unexcused absences; 
whether high school regular participants are more likely to successfully earn the credit they attempted 
in each subject area; and, whether regular participants in 1 st through 3 rd grade are more likely to read at 
grade level. 
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Table B2. Estimated Odds Ratio from Logistic Regression: Regular Participants vs. Comparison Students 




Estimated Odds Ratio 69 

Out-of-school Suspension 

Elementary 

1.47 + 


Middle 

1.07 


High 

0.57 

10+ unexcused absences 

Elementary 

0.66* 


Middle 

0.86 


High 

0.80 

Earning credit attempted (9-12 th grade) 

ELA 

2.43 


Math 

0.42 


Science 

1.23 

DRA: read at grade level 

(l-3rd grade) 

1.12 


tp<o.iO; *p<o.os;**p<o.oi 


69 Odds Ratio (OR) is a measure of association between the treatment and the outcome of interest (e.g. scoring proficient/ advanced in PSSA, 
receiving a C or above) , controlling for the variables that may be predictive of the outcome under study. Specifically, student participation in 
OST programs is not associated with odds of outcome if OR=l; student participation in OST programs is associated with higher odds of 
outcome if 0R>1; and, student participation in OST programs is associated with lower odds of outcome if OR<L 
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Appendix C. Analyses of the Relationship between Program Participation 
Levels and Student Outcomes 

The second set of analyses use participant data to examine the relationship between the number of days 
students participate in OST programs and observed academic and behavioral outcomes. All the analyses 
use linear or logistic regression methods, and the variables controlled for in the models include student 
demographic and family characteristics, as well as student performance before joining the program. 
These characteristics are deemed important for student academic and behavioral performance in the 
literature. We also incorporated organization- level elements of quality programming collected from 
staff survey into the analyses. Including these factors that can be manipulated by providers in our 
analyses may have policy implications and can be methodologically important because of the potential 
correlation between these factors and other variables included in the analyses. It is worth noting that 
any results from this set of analyses should not be generalized to the whole student population since 
only data of OST participants was employed in the analyses. Table C1-C4 reports the results from this 
set of analyses. 70 

Table C I . Regression Results from Analysis of PSSA Data 



Elementary (4-5) 

Middle (6-8) 


Reading 

Math 

Reading 

Math 

OST Days 

0.34* 

0.37* 

0.26* 

0.16 

% full-time staff 

-1.42** 

0.11 

-7.20** 

3.17* 

% staff with at least a Bachelor 

1.05 

-0.34 

12.98** 

-4.24* 

% staff being Certified teachers 

1.19 

-1.04 

24.35** 

-3.76 

% staff majoring in STEM 

- 

- 

9.24** 

-3.95* 

% staff saying STEM was program goal 

- 

- 

-11.75** 

2.31* 

% Staff saying a strong academic was 
program goal 

-2.04** 

-0.54 

4.02** 

-3.87** 

% staff using curriculum for activity 
planning 

-5.28 + 

-1.40 

-30.96** 

5.17 

% staff meeting teachers weekly 

1.65** 

0.26 

7.23** 

0.58 

% staff meeting principals weekly 

-2.57 

0.41 

-20.27** 

1.71 

Cohort 6A(STEM) 

-15.65 

-12.45 

10.43 

1.87 


tf><0. 10; *£><0.05; **£><0.01 


70 Student demographic and background characteristics (i.e., rac^ ethnicity, socioeconomic status, gender, English proficiency levels, disability 
status, and grade levels) and previous year performance were controlled in the analyses. Standard errors were clustered at the provider level. 
Some variables were omitted from analyses because there was little variation across providers or due to issues of missing data and colinearity. 
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Table C2. Regression Results from Analysis of Course Grades Data 



Elementary (4-5) 

Middle (6-8) 


Reading 

Math 

Science 

Reading 

Math 

Science 

OST Days 

0.02** 

0.01 

0.01 

0.02** 

0.02** 

0.01* 

% full-time staff 

0.04** 

-0.01 

0.08** 

0.21** 

0.19* 

0.25** 

% staff with Bachelor or higher ed. levels 

0.16** 

-0.01 

0.09** 

-0.28** 

-0.26 + 

-0.39** 

% staff being certified teachers 

-0.04 

0.04 

-0.17* 

-0.32* 

-0.23 

-0.55* 

% staff majoring in Stem 

-- 

-- 

- 

-0.20** 

-0.29* 

-0.30* 

% staff saying stem was program goal 

- 

- 

- 

-0.01 

-0.03 

0.22** 

% staff saying academic was program goal 

0.29** 

0.06* 

0.44** 

0.03 

-0.08 

-0.26*** 

% staff using curriculum for activity 
planning 

0.17* 

-0.03 

0.39** 

0.67** 

0.46 

0.66* 

% staff meeting teachers weekly 

-0.31** 

-0.01 

-0.40** 

-0.30** 

-0.19* 

-0.12 + 

% staff meeting principals weekly 

-0.09** 

0.01 

-0.02 

0.46** 

0.34 + 

0.30 

Cohort 6A (STEM) 

0.25 

1.46** 

-0.26 

-0.35 

0.23 

-1.53 


tp<0. 10; *f><0.05; **p<0.0. Analysis for elementary science course grades only include 2 nd through 5 th grade. 


Table C3. Results from Logistic Regression; High School Credit (9th through 12th grade) and DRA (l st -3 rd grade) 



Earning Credits attempted 

Reading at grade level (1-3) 


ELA 

Math 

Science 

OST Days 

1.04* 

1.01** 

1.01 

1.01** 

% full-time staff 

0.99 

1.00 

0.94* 

1.00 

% staff with Bachelor or higher 

1.46** 

1.43** 

1.47** 

1.00 

% staff being certified teachers 

1.05** 

1.05** 

1.00 

0.90** 

% using curriculum for activity planning 

-- 

- 

- 

1.10** 

Cohort 6A (STEM) 

0.97 

0.87 

1.56 

1.67** 


tp<O.IO; *p< 0.05; **p<0.0l. 
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Table C4. Results from Logistic Regression: Out-of-School Suspension and Unexcused Absences 



Elementary (2-5) 

Middle ( 6 - 8 ) 

High | 

(9-12) 


10+ Unexcused 
Absences 

Out-of-school 

Suspension 

10+ Unexcused 
Absences 

Out-of-school 

Suspension 

10+ Unexcused 
Absences 

Out-of-school 

Suspension 

OST Days 

0.99** 

0.99** 

0.98** 

0.99** 

0.98** 

0.98** 

% full-time staff 

1 . 02 ** 

0.99 

1.00 

0.99 

0.93** 

0.98** 

% staff with Bachelor or 
higher 

0.95** 

1.01 

1.01 

0.99 

^ 44 ** 

1.67** 

% staff being Certified 
teachers 

1.34** 

0.94* 

1.01 

0.98* 

1.04** 

1.07** 

% staff saying attendance 
as a program goal 

0.97 

- 

1.01 

- 

- 


% staff having access to 
school data 

1.07** 

1.00 

1.01 

1.00 

- 

- 

% staff meet teachers 
weekly 

1.05** 

1 . 02 ** 

0.99 

1.00 

- 

- 

% staff meet principal 
weekly 

1.09** 

0.96** 

1.03** 

1.02 

- 

- 

Cohort 6 a(STEM) 

0.33** 

1.18 

0.46* 

1.11 

0.58 

1.50 + 


tp<O.IO; *p< 0.05; **p<0.0 1 . 


50 





Appendix D. Results from Analyses using Catholic School Data 

RFA also conducted a series of analyses using data collected from the seven 21st CCLC sites that served 
Catholic school students. Like the analyses using public school data, our analyses of Catholic school 
data include two sets of analyses. The first set of analyses examined whether students who regularly 
attended OST programs also attended more days of school and, in turn, achieved better academic 
outcomes than the comparison students. The second set of analyses examines how the OST 
participation levels were related to participants' school attendance and observed academic performance 
on TerraNova and course grades. Table D1 reports the statistical results from analyses of academic 
performance. The predictor for the comparative analyses is a dummy variable indicating whether the 
student regularly participated in the OST program in 20 12- 13. The predictor for the dosage analyses is 
the total days students participated in the OST programs. The outcomes are student numerical course 
grades and TerraNova percentile score in 20 12- 13. 

Table Dl. Statistical Results from Analyses of Academic Performance Data 


Coefficients from comparative analyses: Regular Participants versus comparison students 

TerraNova (2-8th) 

Course Grades (5-8th) 

Reading 

Math 

Reading 

Math 

2.54 

1.00 

-1.35* 

0.22 

Coefficients from dosage analyses: academic performance by OST participation levels 

Reading 

Math 

Reading 

Math 

0.01 

0.02 

-0.01 

0.01 


tp<0. 10; *p<0.05; **p<0.0l. Student previous year performance and indicators of grade levels were controlled in the analyses. 


After controlling for students previous year's performance, our analyses suggests: 

• There is no evidence showing OST regular participants did better in reading and math on 
TerraNova or course grades than comparison students. 

o OST regular participants underperformed comparison students in course grades in 
reading. 

• Among students who attended the programs, there is no sig nifi cant, relationship between OST 
participation levels and student performance on TerraNova or course grades in both reading and 
math. 

Table D2 reports the statistical results from logistic regression. The predictor for comparative analyses 
is a dummy variable indicating whether a student regularly participated in the OST programs in 20 12- 
13. The predictor for the dosage analyses is the total days a student participated in the OST programs. 
The outcome is a dummy variable indicating whether students had 10 or more absences in 20 12- 13. 


Table D2 VMM. Statistical Results from Analyses of School Attendance Data. 




Estimated Odds Ratio 

Comparative Analyses: 

Elementary (2-5) 

0.54** 

Regular Participants vs. Comparison Students 

Middle (6-8) 

0.52** 

Dosage Analyses: 

Elementary (2-5) 

0.62 

Academic Performance by OST Participation Levels 

Middle (6-8) 

0.60 


tp<0. 10; *p<0.05; **p<0.0l. Student previous school absences and dummy variables of grade levels were controlled in the analyses. 
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After controlling for student previous year absences, our analyses suggest: 


• OST regular participants were less likely to have 10 or more absences than comparison students. 

• Among students who attended the program, there is no significant relationship between OST 
participation levels and student likelihood of having 10 or more absences. 

It is worth note that these findings should be interpreted with great caution given the limitations 
associated with the analyses. Besides the limitations discussed in the report, there are other data 
limitations for our analyses of Catholic schools. For example, we were not able to take into account the 
differences in student background characteristics because such information was either not available or 
incomplete in the data we received, whereas literature has suggested that these characteristics are 
important predictors for student academic performance. Furthermore, course grades were in different 
scales across different age grade levels (e.g., 1 st - 3 rd grades reported letter grades while 4 th - 8 th grades 
reported numerical grades), which created challenges in including all students into the analyses. 
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Appendix E. Provider Profile 

Table El. Provider Profiles 


Provider 

Type of Organization 

Cohort 6 
Grantee 

Cohort 6A 
Grantee 

Total number 
of sites 

Catholic Social Services 

Non-profit Multi-Service 
Organization 

Yes 


10 

Education Works 

Non-Profit organization 
Education and Youth 
Development Organization 

Yes 

Yes 

6 

Public Health Management 
Corporation 

Intermediary Organization 

Yes 


4 

Pan American Academy 
Charter School 

Charter School 

Yes 


1 

Netter Center for 
Community Partnerships at 
the University of 
Pennsylvania 

University-based non-profit 
organization 

Yes 


3 

Congreso de Latinos Unidos 

Non-profit organization Multi- 
Service Organization 

Yes 

Yes 

8 

Philadelphia Arts and 
Education Partnership 

Non-profit Arts Education 
organization 


Yes 

4 

To Our Children's Future 
with Health 

Non-profit, Education and 
Youth Development 
Organization 


Yes 

2 

Sunrise 

Non-profit Education and Youth 
Development Organization 


Yes 

1 

Communities in Schools 

National non-profit 
Multi-service organization 


Yes 

4 

City Year of Greater 
Philadelphia 

National non-profit 
Education and Youth 
Development Organization 


Yes 

6 


53 


